Skip to content

Speak typed characters in Windows consoles in the correct language (second try) - #10554

Merged
michaelDCurran merged 3 commits into
masterfrom
i10113_winEvents
Nov 28, 2019
Merged

michaelDCurran merged 3 commits into
masterfrom
i10113_winEvents

Conversation

@michaelDCurran

Copy link
Copy Markdown
Member

Replaces pr #10551

Link to issue number:

Fixes #10113

Summary of the issue:

With the work to support Windows consoles via UIA, NVDA moved to speaking typed characters in consoles by detecting keyboard input and then using ToUnicodeEx to convert the keyboard input into a character for speaking. We had already been taking this approach for Microsoft Edge and other places in Windows 10 where we could not inject in-process.
ToUnicodeEx requires a keyboard layout to use the correct language. Therefore, we would call GetKeyboardLayout, giving it the thread ID of the focused window.
However, it seems that Windows console windows report the wrong thread ID, and therefore, GetKeyboardLayout fails, returning 0 instead of a valid keyboard layout.
Passing this 0 to ToUnicodeEx, it would generate characters in the user's system language (I think). At least, on my system it was always english US no matter what input language I was currently using.

Description of how this pull request fixes the issue:

Override the windowThreadID property on both UIA and legacy WinConsole NVDAObjects, to reflect the true thread ID of the window. This thread ID is found in a new cache of console windows to thread IDs in IAccessibleHandler, filled in by received winEvents for ConsoleWindowClass windows.
Received winEvents always contain an ID of the true thread that NotifyWinEvent was called in. In the case of ConsoleWindowClass winEvents, this is from inside conhost.
this thread happens to be the true input (focus thread) for consoles, and therefore GetKeyboardLayout returns the correct input language for the console, when given this threadID.
We can garantee that there will at least either be a focus winEvent, or a locationChange winEvent (for the caret) for consoles, when moving to a console and or starting NVDA within a console. This is enough to make sure that we have a threadID for that console window when we need it.

Testing performed:

  • Installed the Arabic keyboard layout.
  • Opened a Windows console and ensured NVDA was using UIA.
  • With speak typed characters on, first typed some characters, confirming they were being announced in english. backspacing them also confirming they were english characters written to the console.
  • Switched keyboard layouts to Arabic.
  • Again typed some characters, confirming that NVDA was speaking them as Arabic. Backspacing them, confirmed that actual arabic characters were written to the console.

Known issues with pull request:

It is possible that the very first typed character when starting NVDA from within a console will be announced in the wrong language, as the caret has not yet moved once. But this is rather rare.

Change log entry:

None

… saving them for exposia as windowThreadID in WinConsole NVDAObjects. Necessary for speak typed characters. Addresses #10113.
@AppVeyorBot

Copy link
Copy Markdown

PR introduces Flake8 errors 😲

See test results for Failed build of commit 712c583a82

@codeofdusk codeofdusk left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks fine to me, assuming that nothing in the legacy console support depends on the previous threadID behavior.

@LeonarddeR LeonarddeR left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only some style comments

Comment thread source/IAccessibleHandler.py Outdated
# But NVDA really requires to know the actual thread the window was created in,
# I.e. inside conhost,
# In order to handle speaking of typed characters etc.
# winEventCallback adds these when ever it sees an event for ConsoleWindowClass windows,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be whenever?

Suggested change
# winEventCallback adds these when ever it sees an event for ConsoleWindowClass windows,
# winEventCallback adds these whenever it sees an event for ConsoleWindowClass windows,

Comment thread source/IAccessibleHandler.py Outdated
# In order to handle speaking of typed characters etc.
# winEventCallback adds these when ever it sees an event for ConsoleWindowClass windows,
# As winEvents always contain the true thread ID.
consoleWindowsToThreadIDs = {}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typing

Suggested change
consoleWindowsToThreadIDs = {}
consoleWindowsToThreadIDs: Dict[int, int] = {}

#: the caret in consoles can take a while to move on Windows 10 1903 and later.
_caretMovementTimeoutMultiplier = 1.5

def _get_windowThreadID(self):

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not that I think this should happen now, but I wonder whether we want a base class for console windows that the legacy and UIA impls inherit from.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

UIA console: when typing in the console, characters are wrongly pronounced regardless of the keyboard layout

5 participants